Image processing apparatus, image processing method, and program

ABSTRACT

An in-image capturing unit captures an image of a subject in a direction facing a front surface of an image processing apparatus, as an in-image, and an out-image capturing unit captures an image of a subject in a direction facing a back surface of the image processing apparatus, as an out-image. A signal processing unit obtains a binary image as additional information, the binary image being obtained by binarizing the in-image, for example, and adds the additional information to image data of the out-image. Additionally, the signal processing unit combines the binary image serving as the additional information with the out-image when the out-image is reproduced, and displays the resultant image. In such a manner, information obtained from the in-image is added to the out-image as additional information, and thus it is possible to more effectively display an image and improve convenience.

TECHNICAL FIELD

The present technology relates to an image processing apparatus, animage processing method, and a program, and particularly to an imageprocessing apparatus, an image processing method, and a program that arecapable of displaying an image more effectively and also improvingconvenience.

BACKGROUND ART

Heretofore, there have been known imaging apparatuses each equipped withtwo cameras. Some of such imaging apparatuses can simultaneously captureimages with the two cameras.

Examples of such imaging apparatuses include an imaging apparatus thatcaptures images with a main camera and a sub-camera, and when acquiringvoice of a photographer by a sub-microphone, combines an image obtainedby the sub-camera with an image obtained by the main camera, forrecording (see, for example, Patent Document 1).

Patent Document 1: Japanese Patent Application Laid-open No. 2011-250340

SUMMARY OF INVENTION Problem to be Solved by the Invention

In the technology described above, however, after the image obtained bythe sub-camera is combined with the image obtained by the main camera,this makes it impossible to display only an image obtained by the maincamera and not subjected to image composition. As a result, capturedimages cannot be displayed fully effectively.

Additionally, when switching on whether to perform image composition ornot is intended to be performed, instructions should be given byutterance or the like each time an image is captured, which isburdensome. As a result, a photographer cannot concentrate on capturingimages in some cases.

The present technology has been made in view of such circumstances andmakes it possible to display an image more effectively and also improveconvenience.

Means for Solving the Problem

According to one aspect of the present technology, there is provided animage processing apparatus, including: a first imaging unit that imagesa first direction; a second imaging unit that images a second direction,the second direction being different from the first direction; anadditional-information generating unit that generates additionalinformation based on a first image, the first image being obtained inimaging by the first imaging unit; and an additional processing unitthat adds the additional information to image data of a second image,the second image being obtained in imaging by the second imaging unit.

The additional-information generating unit may compress an informationamount of the first image, to generate the additional information.

The additional-information generating unit may compress the size of thefirst image, to generate the additional information.

The first imaging unit may image a direction of a user operating theimage processing apparatus, the direction serving as the firstdirection, and the image processing apparatus may further include acombining unit that combines an image obtained from the additionalinformation with the second image.

The first imaging unit may image a direction of a user operating theimage processing apparatus, the direction serving as the firstdirection, and the additional-information generating unit may performface recognition processing on the first image, to generate photographerinformation of the second image as the additional information.

The image processing apparatus may further include: an extraction unitthat detects a second image from the plurality of second images based onthe additional information, the second image being captured by aspecific photographer; and a reproduction unit that displays only thesecond image detected by the extraction unit.

The image processing apparatus may further include a classification unitthat records the second image in a recording area determined by thephotographer information, the photographer information serving as theadditional information.

The image processing apparatus may further include a display unit thatis provided to a surface facing a user operating the image processingapparatus, the first imaging unit being provided to the surface, anddisplays an image taken in by the second imaging unit at a time when thesecond image is captured, in which the additional-information generatingunit detects a line-of-sight direction of the user based on the firstimage, to generate information indicating a gaze area at which the usergazes on the second image, the information serving as the additionalinformation.

The image processing apparatus may further include a reproduction unitthat controls display of the second image based on the additionalinformation.

The reproduction unit may superimpose and display the plurality ofsecond images based on the additional information.

The image processing apparatus may further include a viewing-angleconverting unit that performs viewing-angle conversion processing on thesecond image, in which the additional-information generating unit mayobtain, as the additional information, a cropped image obtained bycutting out an area including a main subject on the first image, and theadditional processing unit may add the additional information to theimage data of the second image on which the viewing-angle conversionprocessing is performed.

The image processing apparatus may further include a combining unit thatarranges and combines the cropped image serving as the additionalinformation with the second image on which the viewing-angle conversionprocessing is performed, to generate a composite image with an aspectratio that is the same as an aspect ratio of the second image capturedby the second imaging unit.

The first imaging unit may image a direction of a user operating theimage processing apparatus, the direction serving as the firstdirection, the additional-information generating unit may detect a smileof the user based on the first image and generate the degree of smile ofthe user as the additional information, and the second imaging unit mayimage the second direction when the smile of the user is detected.

The image processing apparatus may further include a classification unitthat records the second image in a recording area determined by thedegree of smile, the degree of smile serving as the additionalinformation.

The first imaging unit may image a direction of a user operating theimage processing apparatus, the direction serving as the firstdirection, the additional-information generating unit may detect a smileof the user based on the first image and generate the degree of smile ofthe user as the additional information, and the image processingapparatus may further include a classification unit that records thesecond image in a recording area determined by the degree of smile, thedegree of smile serving as the additional information.

According to one aspect of the present technology, there is provided animage processing method or a program, including: imaging, by a firstimaging unit, a first direction; imaging, by a second imaging unit, asecond direction that is different from the first direction; generatingadditional information based on a first image that is obtained inimaging by the first imaging unit; and adding the additional informationto image data of a second image that is obtained in imaging by thesecond imaging unit.

In one aspect of the present technology, a first direction is imaged bya first imaging unit, a second direction that is different from thefirst direction is imaged by a second imaging unit, additionalinformation is generated based on a first image that is obtained inimaging by the first imaging unit, and the additional information isadded to image data of a second image that is obtained in imaging by thesecond imaging unit.

Effect of the Invention

According to an aspect of the present technology, it is possible todisplay an image more effectively and also improve convenience.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a configuration example of the outerappearance of an image processing apparatus.

FIG. 2 is a diagram for describing display of an out-image.

FIG. 3 is a diagram showing a configuration example of the imageprocessing apparatus.

FIG. 4 is a diagram showing a configuration example of a signalprocessing unit.

FIG. 5 is a flowchart for describing imaging processing.

FIG. 6 is a diagram for describing a run length code.

FIG. 7 is a flowchart for describing reproduction processing.

FIG. 8 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 9 is a flowchart for describing imaging processing.

FIG. 10 is a flowchart for describing reproduction processing.

FIG. 11 is a diagram showing a display example of a list of out-images.

FIG. 12 is a diagram showing a display example of a list of out-images.

FIG. 13 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 14 is a flowchart for describing imaging processing.

FIG. 15 is a flowchart for describing reproduction processing.

FIG. 16 is a diagram showing a display example of out-images in a slideshow.

FIG. 17 is a diagram for describing a composite image.

FIG. 18 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 19 is a flowchart for describing imaging processing.

FIG. 20 is a flowchart for describing reproduction processing.

FIG. 21 is a diagram for describing recording of out-images on aphotographer basis.

FIG. 22 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 23 is a flowchart for describing imaging processing.

FIG. 24 is a flowchart for describing reproduction processing.

FIG. 25 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 26 is a flowchart for describing imaging processing.

FIG. 27 is a flowchart for describing reproduction processing.

FIG. 28 is a diagram for describing recording of out-images on adegree-of-smile basis.

FIG. 29 is a diagram showing a configuration example of the signalprocessing unit.

FIG. 30 is a flowchart for describing imaging processing.

FIG. 31 is a flowchart for describing reproduction processing.

FIG. 32 is a flowchart for describing imaging processing.

FIG. 33 is a diagram showing a configuration example of a computer.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments to which the present technology is applied willbe described with reference to the drawings.

First Embodiment

<Regarding the Overview of the Present Technology>

The present technology relates to a device including a plurality ofimaging units that are capable of imaging mutually different directions,such as multi-function mobile phones and digital cameras. Such a deviceacquires information from an image captured simultaneously with anotherimage when the other image is captured by a main imaging unit, togenerate the information as additional information. In the presenttechnology, such additional information is generated, and thus imagescan be displayed more effectively. Additionally, the additionalinformation is embedded in image data, and thus display control with ahigher degree of freedom can be performed at the time when images arereproduced, without requiring burdensome operations at the time whenimages are captured, and convenience can be improved.

FIG. 1 is a diagram showing a configuration example of the outerappearance of an image processing apparatus such as a multi-functionmobile phone to which the present technology is applied.

In FIG. 1, an outer appearance of an image processing apparatus 11indicated by an arrow A11 is an outer appearance on a front surfaceside, that is, on a side viewed from a user when the user holds andoperates the image processing apparatus 11. Additionally, an outerappearance of the image processing apparatus 11 indicated by an arrowA12 is an outer appearance on a side surface side, that is, an outerappearance when the image processing apparatus 11 indicated by the arrowA11 is viewed from the left side or the right side of the drawing.Further, an outer appearance of the image processing apparatus 11indicated by an arrow A13 is an outer appearance on a rear surface side,that is, on the opposite side to the front surface side.

As indicated by the arrow A11, a display unit 21 that displays an imageor the like is provided on the front surface side of the imageprocessing apparatus 11. A touch panel used for the user to perform aninput operation is provided to be superimposed on the display unit 21.Additionally, in the drawing, an in-image capturing unit 22 is providedon the upper side of the display unit 21 on the front surface of theimage processing apparatus 11. The in-image capturing unit 22 capturesan image of a subject present in a direction of the front surface sideof the image processing apparatus 11. Therefore, an image captured bythe in-image capturing unit 22 is an image of the user operating theimage processing apparatus 11 in most cases. Hereinafter, an imagecaptured by the in-image capturing unit 22 is referred to as anin-image. The description will be continued assuming that an image ofthe user is captured as the in-image.

Further, as indicated by the arrow A13, an out-image capturing unit 23is provided on the rear surface of the image processing apparatus 11.The out-image capturing unit 23 captures an image of a subject presentin a direction of the rear surface side of the image processingapparatus 11. In this example, the out-image capturing unit 23 isprovided to capture an image in the opposite direction to the imagingdirection of the in-image capturing unit 22. However, the in-imagecapturing unit 22 and the out-image capturing unit 23 only need to bedisposed such that imaging directions thereof are different from eachother.

An image captured by the out-image capturing unit 23 is normally animage of a landscape, a human, or the like that the user is viewing,that is, an image of a subject that the user considers desirable toimage. Hereinafter, an image captured by the out-image capturing unit 23is referred to as an out-image.

In the case where the user captures an image of a desired subject withsuch an image processing apparatus 11, as shown in the left side of FIG.2, for example, a user U11 directs the out-image capturing unit 23 ofthe image processing apparatus 11 toward a target subject H11 and givesan instruction to capture images. The image processing apparatus 11 thencaptures an in-image and an out-image simultaneously or almostsimultaneously, according to the instruction to capture images by theuser U11.

Therefore, in the image processing apparatus 11, an image of the subjectH11 is obtained as the out-image, and an image of the user U11 isobtained as the in-image.

The image processing apparatus 11 performs processing such as encodingon the in-image thus obtained, as necessary, and obtains the resultantimage as additional information. The image processing apparatus 11 thenadds the obtained additional information to image data of the out-image.

Here, the additional information is not combined on the out-image, butis embedded as additional information in the image data of theout-image. Specifically, the additional information is embedded in theimage data of the out-image conforming to, for example, Exif(Exchangeable image file format), as so-called Exif data, that is, asmetadata on the out-image.

Additionally, at the time when the out-image is reproduced, the user canselect whether to display only the out-image or whether to combine animage obtained from the additional information with the out-image fordisplay. For example, in the case where the image obtained from theadditional information is combined with the out-image for display, anout-image shown on the right side in the figure is displayed on thedisplay unit 21.

In this example, an image IP11 obtained from the additional informationis superimposed and displayed on the out-image in which the subject H11appears.

Since the image IP11 is an image of the face of the user U11, a userviewing the out-image can know a person who has captured the out-image,a facial expression of the user U11 when capturing the out-image, thatis, whether the user is smiling, angry, excited, or the like.

In such a way, the image obtained from the in-image is embedded in theimage data of the out-image as additional information, and theadditional information is used as necessary at the time when theout-image is reproduced. This makes it possible to check a situation ofa photographer when capturing the out-image. Consequently, whenout-images captured during a trip, for example, are viewed later, it ispossible to enjoy more the out-images as memories.

Additionally, according to the image processing apparatus 11, since asituation where the out-image is captured can be checked in the image,for example, if an out-image is captured as reference information at thetime when an accident is investigated, information on a photographer, animaging environment, an imaging location, and the like can be easilysaved.

(Configuration Example of Image Processing Apparatus)

Next, a more detailed configuration of the image processing apparatus 11will be described.

FIG. 3 is a block diagram showing a more detailed configuration exampleof the image processing apparatus 11. In FIG. 3, the same referencenumerals are given to portions corresponding to the portions in FIG. 1and description thereof will be appropriately omitted.

The image processing apparatus 11 shown in FIG. 3 includes the displayunit 21, the in-image capturing unit 22, the out-image capturing unit23, an operation input unit 51, a control unit 52, a signal processingunit 53, and a recording unit 54.

The operation input unit 51 is formed by, for example, a touch panelprovided to be superimposed on the display unit 21 and supplies a signalcorresponding to a user's operation to the control unit 52.

The control unit 52 controls the entire operation of the imageprocessing apparatus 11 according to the signal supplied from theoperation input unit 51. For example, the control unit 52 instructs thein-image capturing unit 22 or the out-image capturing unit 23 to capturean image or instructs the signal processing unit 53 to reproduce orrecord an out-image.

The signal processing unit 53 generates additional information based onan in-image supplied from the in-image capturing unit 22. The signalprocessing unit 53 adds the additional information to an out-imagesupplied from the out-image capturing unit 23 and also supplies theout-image, to which the additional information is added, to therecording unit 54, to record the out-image therein. Additionally, thesignal processing unit 53 reads the out-image from the recording unit 54and supplies the out-image to the display unit 21 for display.

The recording unit 54 records the out-image supplied from the signalprocessing unit 53 and also supplies the recorded out-image to thesignal processing unit 53 as necessary.

<Configuration Example of Signal Processing Unit>

The signal processing unit 53 of FIG. 3 is configured as shown in FIG. 4in more detail.

The signal processing unit 53 shown in FIG. 4 includes a compressionunit 81, an additional processing unit 82, an extraction unit 83, adecoding unit 84, and a reproduction unit 85.

The compression unit 81 compresses the amount or size of information onthe in-image supplied from the in-image capturing unit 22 to generateadditional information, and supplies the additional information to theadditional processing unit 82. The compression unit 81 includes aninformation-amount compressing unit 91, a size compressing unit 92, andan encoding unit 93.

The information-amount compressing unit 91 binarizes the in-image, toextract the outline of a subject on the in-image. In other words, thein-image is converted into a binary image by binarization processing,and this leads to a compression of the information amount of thein-image.

The size compressing unit 92 reduces the size of the binary imageobtained in the information-amount compressing unit 91, to compress thesize of the binary image. The encoding unit 93 encodes the binary imagehaving the size compressed by the compression unit 92, to obtain thebinary image as additional information. It should be noted that theencoding processing by the encoding unit 93 can also be processing ofcompressing the information amount of the in-image (binary image).

The additional processing unit 82 adds the additional information, whichis supplied from the compression unit 81, to the image data of theout-image supplied from the out-image capturing unit 23, and suppliesthe resultant out-image to the recording unit 54.

The extraction unit 83 reads the out-image from the recording unit 54and extracts the additional information from the out-image.Additionally, the extraction unit 83 supplies the out-image to thereproduction unit 85 and also supplies the additional information, whichis extracted from the out-image, to the decoding unit 84.

The decoding unit 84 decodes the additional information supplied fromthe extraction unit 83 and supplies the resultant binary image to thereproduction unit 85. Additionally, the decoding unit 84 includes adecompression unit 94. The decompression unit 94 decompresses the sizeof the binary image as necessary.

The reproduction unit 85 supplies the out-image, which is supplied fromthe extraction unit 83, to the display unit 21 for display.Additionally, the reproduction unit 85 includes a combining unit 95. Thecombining unit 95 combines the binary image supplied from the decodingunit 84 with the out-image. Therefore, the reproduction unit 85 candisplay the out-image, with which the binary image is combined asnecessary, on the display unit 21.

<Description on Imaging Processing>

Incidentally, when the user performs operations with respect to theoperation input unit 51 to select a mode in which additional informationis added to an out-image and to give an instruction to capture an imageof a desired subject, the image processing apparatus 11 starts imagingprocessing and records an out-image in which additional informationembedded. Hereinafter, the imaging processing performed by the imageprocessing apparatus 11 will be described with reference to theflowchart of FIG. 5.

In Step S11, the out-image capturing unit 23 captures an out-imageaccording to an instruction of the control unit 52 and supplies theout-image to the additional processing unit 82. For example, a landscapeimage is captured as the out-image.

In Step S12, the in-image capturing unit 22 captures an in-imageaccording to an instruction of the control unit 52 and supplies thein-image to the compression unit 81. For example, an image of the faceof the user operating the image processing apparatus 11 is captured asthe in-image.

In Step S13, the information-amount compressing unit 91 converts thein-image supplied from the in-image capturing unit 22 into a binaryimage. For example, the information-amount compressing unit 91 convertsthe in-image as a color image into a monochrome image in 256 gradationlevels, and further binarizes the monochrome image to generate a binaryimage.

Consequently, for example, the image of the face of the user whocaptures the out-image is obtained as a binary image. Since this binaryimage is an image of the face of the user when the out-image iscaptured, a facial expression, a feeling, and the like of the user whenthe out-image is captured can be known from the image.

In Step S14, the size compressing unit 92 compresses the size of thebinary image obtained in the information-amount compressing unit 91. Forexample, the binary image having the same size as the out-image isreduced to the size of 160×120 pixels, that is, to a thumbnail size ofthe image.

In Step S15, the encoding unit 93 encodes the binary image having thesize compressed by the compression unit 92 and obtains the binary imageas additional information.

For example, as shown in FIG. 6, the encoding unit 93 performs encodingon a binary image RP11 by run length encoding, and the resultant runlength code is obtained as additional information.

It should be noted that each square of the binary image RP11 representsone pixel. In particular, a white square represents a pixel having apixel value of “1”, and a black square represents a square having apixel value of “0”.

Additionally, in FIG. 6, coordinates that indicate positions in ahorizontal direction of the binary image RP11 in the figure are shown.For example, coordinates of pixels positioned at the left end of thebinary image RP11 in the figure are represented as “1”, and coordinatesof pixels positioned at the right end in the figure are represented as“11”. In other words, the coordinates of the respective pixels representat which number those pixels are positioned from the left end of thebinary image RP11.

In the case where the binary image RP11 is encoded by run lengthencoding, the binary image RP11 is encoded on a pixel-row basis, thepixel rows each having pixels arranged in the horizontal direction inthe figure. In the encoding of the pixel rows, a pixel row to be encodedis expressed by a position of a starting point of the pixel row and arun length code. The pixel row is formed of pixels each having a pixelvalue of “1”. The run length code is formed by a length of the pixelrow.

Specifically, for example, assuming that the uppermost pixel row of thebinary image RP11 in the figure is a pixel row of interest, in thispixel row of interest, a pixel at a coordinate of “1” has a pixel valueof “1”, and a pixel at a coordinate of “2” adjacent thereto has a pixelvalue of “0”. Therefore, assuming that a pixel row having a startingpoint at a coordinate of “1” and including a pixel having a pixel valueof “1” is expressed by “starting point, length” of the pixel row, “1,1”is obtained.

In the same manner, in the pixel row of interest, assuming that a pixelrow having a starting point at a coordinate of “4” and including pixelseach having a pixel value of “1” is expressed by “starting point,length”, “4,3” is obtained. Further, in the pixel row of interest,assuming that a pixel row having a starting point at a coordinate of“10” and including pixels each having a pixel value of “1” is expressedby “starting point, length”, “10,2” is obtained.

In this regard, as shown on the right side of the figure, the encodingunit 93 sets a code “1,1,4,3,10,2”, which is obtained by arranging thecodes indicating those three sets of “starting point, length”, as a runlength code of the pixel row of interest. The encoding unit 93 sets therun length code thus obtained for each pixel row of the binary imageRP11 as additional information of the binary image RP11.

The compression unit 81 supplies the additional information obtained byencoding by the encoding unit 93 to the additional processing unit 82.

Referring back to the description on the flowchart of FIG. 5, in StepS16, the additional processing unit 82 adds the additional information,which is supplied from the compression unit 81, to the image data of theout-image supplied from the out-image capturing unit 23, and suppliesthe resultant data to the recording unit 54. In other words, theadditional information is embedded in the image data of the out-image,to serve as metadata of the out-image.

In Step S17, the recording unit 54 records the out-image supplied fromthe additional processing unit 82, and then the imaging processing isterminated.

As described above, the image processing apparatus 11 compresses theinformation of the in-image captured simultaneously with the out-imageto covert the information into additional information, and adds theadditional information to the out-image. In such a manner, theadditional information is added to the out-image, and thus it ispossible to display an image more effectively and also improveconvenience using the additional information at the time when theout-image is reproduced.

For example, in the image processing apparatus 11, the additionalinformation is added to the out-image, and thus whether to display onlythe out-image or whether to display the out-image using the additionalinformation can be specified at the time of reproduction. Therefore, itis possible to display the out-image more effectively and also improveconvenience.

In addition, when capturing the out-image, the user does not need anoperation to give an instruction to combine an image obtained from thein-image with the out-image, and can thus concentrate on an imagingoperation. Consequently, user's convenience can further be improved.

<Description on Reproduction Processing>

When the out-image is recorded in the recording unit 54, the user canoperate the operation input unit 51 to reproduce the recorded out-image.At that time, the user operates the operation input unit 51, forexample, and can thus select whether to display only the out-image orwhether to combine an image, which is obtained from the additionalinformation, with the out-image for display.

Hereinafter, with reference to a flowchart of FIG. 7, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to reproduce theout-image.

In Step S41, the extraction unit 83 reads the out-image, which isspecified by the user according to an instruction of the control unit52, from the recording unit 54.

In Step S42, the extraction unit 83 supplies the read out-image to thereproduction unit 85, and also reads additional information from theimage data of the out-image to supply the additional information to thedecoding unit 84.

In Step S43, the decoding unit 84 decodes the additional informationsupplied from the extraction unit 83. For example, in the case where theadditional information is a run length code, the decoding unit 84restores the binary image from the run length code.

In Step S44, the decompression unit 94 decompresses the size of thebinary image obtained by decoding. For example, a binary image having athumbnail size of 160×120 pixels is enlarged to be a binary image havingan appropriate size.

The decoding unit 84 supplies the binary image, which is decompressed bythe decompression unit 94, to the reproduction unit 85 as the imageobtained from the additional information.

In Step S45, the reproduction unit 85 determines whether to reproducethe binary image.

For example, in the case where the user operates the operation inputunit 51 and gives an instruction to combine the binary image obtainedfrom the additional information with the out-image for display, a signalindicating the instruction is supplied from the control unit 52 to thesignal processing unit 53. In the case where the signal supplied fromthe control unit 52 to the signal processing unit 53 indicates that thebinary image is combined with the out-image for display, thereproduction unit 85 determines that the binary image is reproduced.

When it is determined in Step S45 that the binary image is reproduced,in Step S46, the combining unit 95 combines the binary image, which issupplied from the decoding unit 84, with the out-image supplied from theextraction unit 83. For example, a position on the out-image at whichthe binary image is combined may be a position determined in advance, aposition specified by the user, or a position in a background areawithout a subject (foreground), which is identified by image recognitionor the like.

The reproduction unit 85 supplies the out-image with which the binaryimage is combined to the display unit 21 and controls display of theout-image.

In Step S47, the display unit 21 displays the out-image, which issupplied from the reproduction unit 85 and with which the binary imageis combined, and the reproduction processing is terminated.

Additionally, when it is determined in Step S45 that the binary image isnot reproduced, the reproduction unit 85 supplies the out-image suppliedfrom the extraction unit 83 to the display unit 21 as it is, and theprocessing proceeds to Step S48. In this case, the out-image with whichthe binary image is not combined is supplied to the display unit 21.

In Step S48, the display unit 21 displays the out-image supplied fromthe reproduction unit 85, and the reproduction processing is terminated.

In such a way, the image processing apparatus 11 displays the out-imageas it is according to a user's instruction or combines the binary image,which is obtained from the additional information added to theout-image, with the out-image for display.

If the additional information added to the out-image is used in such amanner, it is possible to display only the out-image in accordance with,for example, a feeling of the user at that time or display the out-imagewith which the binary image is combined. Consequently, it is possible todisplay the out-image more effectively and improve convenience.

In particular, if the binary image is combined with the out-image, auser who views the out-image can know a facial expression of the face ofthe user when capturing the out-image or the like, and can thus enjoyviewing the out-image more.

Hereinabove, the cases where a monochrome conversion, binarization, areduction in size, and encoding of the in-image are performed have beendescribed as an example in which the information of an in-image iscompressed, but any types of those compression processing may beappropriately performed in combination. For example, only the monochromeconversion of the in-image may be performed to obtain additionalinformation, or the monochrome conversion and encoding of the in-imagemay be performed to obtain additional information. Additionally, a partof an area of the in-image may be compressed, and the resultant image orcode may be obtained as additional information.

Second Embodiment

<Configuration Example of Signal Processing Unit>

Although the example in which the binary image obtained from thein-image is generated as additional information has been describedhereinabove, other information in addition to the binary image may begenerated as additional information. For example, the binary image andphotographer information obtained from the in-image may be generated asadditional information.

In this case, the signal processing unit 53 has, for example, aconfiguration shown in FIG. 8. In FIG. 8, the same reference numeralsare given to portions corresponding to the portions in FIG. 4 anddescription thereof will be appropriately omitted.

The signal processing unit 53 shown in FIG. 8 includes a compressionunit 81, an additional processing unit 82, an extraction unit 83, adecoding unit 84, a reproduction unit 85, and a face recognition unit121. In other words, the signal processing unit 53 of FIG. 8 has aconfiguration in which the signal processing unit 53 of FIG. 4 furtherincludes the face recognition unit 121.

The face recognition unit 121 performs face recognition on an in-imagesupplied from the in-image capturing unit 22 by using registeredinformation recorded in advance in the recording unit 54, and supplies aresult of the recognition to the additional processing unit 82 asadditional information.

For example, in the recording unit 54, a feature amount of an image ofthe face of a user registered in advance, a discriminator of the face ofthe user, and the like are associated with a user name of the user, andthe associated information is recorded as registered information. Theface recognition unit 121 identifies the user on the in-image by facerecognition using the registered information and the in-image, and setsthe result of the identification (the result of the face recognition) asadditional information.

The additional processing unit 82 sets, as additional information, theresult of the face recognition supplied from the face recognition unit121 and the binary image supplied from the compression unit 81, and addsthe additional information to the out-image from the out-image capturingunit 23, to supply the resultant out-image to the recording unit 54.

<Description on Imaging Processing>

Next, with reference to a flowchart of FIG. 9, description will be givenon imaging processing performed when the signal processing unit 53 hasthe configuration shown in FIG. 8.

Since processing of Steps S71 to S75 are similar to the processing ofSteps S11 to S15 of FIG. 5, description thereof will be omitted. In StepS72, however, an in-image obtained by imaging is supplied from thein-image capturing unit 22 to the compression unit 81 and the facerecognition unit 121.

In Step S76, the face recognition unit 121 performs face recognition onthe in-image supplied from the in-image capturing unit 22 usingregistered information recorded in the recording unit 54 and supplies aresult of the recognition as additional information to the additionalprocessing unit 82. For example, the user name of the user identified asa result of the face recognition, that is, photographer information issupplied to the additional processing unit 82 as additional information.

It should be noted that as a result of the face recognition, in the casewhere the user on the in-image is a new user who is not registered or auser whose user name is not recorded as registered information,registration processing and the like are performed as necessary.

In Step S77, the additional processing unit 82 adds the photographerinformation supplied from the face recognition unit 121 and the encodedbinary image supplied from the compression unit 81 to the image data ofthe out-image supplied from the out-image capturing unit 23, asadditional information, and supplies the resultant data to the recordingunit 54.

In Step S78, the recording unit 54 records the out-image supplied fromthe additional processing unit 82, and then the imaging processing isterminated.

As described above, the image processing apparatus 11 adds thephotographer information and the binary image obtained from the in-imageas additional information to the out-image.

Consequently, it is possible to more effectively display the out-imageand also improve convenience. For example, if the photographerinformation generated as additional information is used, it is possibleto effectively display the out-image, for example, to display a list ofout-images captured by a specific photographer.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display a list ofout-images captured by a specific user. At that time, the user operatesthe operation input unit 51, for example, and can thus select whether todisplay only out-images or whether to combine an image, which isobtained from the additional information, with the out-images fordisplay.

Hereinafter, with reference to a flowchart of FIG. 10, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to display a list ofout-images.

In Step S101, the extraction unit 83 reads all out-images recorded inthe recording unit 54.

In Step S102, the extraction unit 83 reads the photographer informationas additional information, which is added to each of the readout-images, and detects an out-image captured by a specificphotographer.

For example, the specific photographer is assumed to be a photographerthat is input by the user operating the operation input unit 51. Thecontrol unit 52 reads the registered information from the recording unit54 via the signal processing unit 53 as necessary, based on a signalfrom the operation input unit 51, and supplies information indicating aphotographer specified by the user to the signal processing unit 53. Asa result, the extraction unit 83 detects an out-image captured by thespecific photographer, based on the information supplied from thecontrol unit 52.

In Step S103, the extraction unit 83 supplies only the detectedout-image of the specific photographer to the reproduction unit 85, andalso reads the encoded binary image as additional information from theimage data of the out-image of the specific photographer, to supply theinformation to the decoding unit 84. For example, a run length code issupplied to the decoding unit 84 as the encoded binary image.

After the additional information is read from the out-image, theprocessing of Steps S104 to S107 are performed. Since those processingare similar to the processing of Steps S43 to S46 in FIG. 7, descriptionthereof will be omitted.

In Steps S104 to S107, however, for all the out-images captured by thespecific photographer, the encoded binary image as additionalinformation read from each of the out-images is decoded, and the binaryimages are combined with the respective out-images.

In Step S108, the display unit 21 displays a list of the out-images,which are supplied from the reproduction unit 85 and with which thebinary images are combined, and the reproduction processing isterminated.

Consequently, out-images SM11-1 to SM11-6 shown in FIG. 11, for example,are displayed on the display unit 21. In this example, the sixout-images SM11-1 to SM11-6 are arranged for display on the display unit21. Binary images IP21-1 to IP21-6 as additional information arecombined with those out-images SM11-1 to SM11-6, respectively, fordisplay.

In such a manner, only the out-images of a specific photographer aredisplayed in a list, and thus the user can more easily find out anout-image that the user wants to view. In other words, it is possible tomore effectively display the out-images.

For example, when the out-images SM11-1 to SM11-6 are displayed in alist, the user can operate the operation input unit 51 to display onlyone of those out-images on the display unit 21 or display thoseout-images in a slide show.

It should be noted that here, the example in which the binary image iscombined with each out-image for display has been described, but animage of the face of the user may be recorded in advance as registeredinformation to be combined with the out-image for display.

Referring back to the description of the flowchart of FIG. 10, when itis determined in Step S106 that the binary images are not reproduced,the reproduction unit 85 supplies the out-images, which are suppliedfrom the extraction unit 83, to the display unit 21 as they are, and theprocessing proceeds to Step S109.

In Step S109, the display unit 21 displays a list of the out-imagessupplied from the reproduction unit 85, and the reproduction processingis terminated.

Consequently, the out-images SM11-1 to SM11-6 shown in FIG. 12, forexample, are displayed on the display unit 21. It should be noted thatin FIG. 12, the same reference symbols are given to portionscorresponding to the case in FIG. 11 and description thereof will beappropriately omitted.

In the example of FIG. 12, the six out-images SM11-1 and SM11-6 arearranged for display on the display unit 21. On those out-images SM11-1to SM11-6, binary images as additional information are not displayed.

In such a manner, only the out-images of a specific photographer aredisplayed in a list, and thus the user can more easily find out anout-image that the user wants to view, as in the example of FIG. 11.

For example, when the out-images SM11-1 to SM11-6 are displayed in alist, the user can operate the operation input unit 51 to display onlyone of those out-images on the display unit 21 or display thoseout-images in a slide show.

As described above, the image processing apparatus 11 detects theout-images of a specific photographer using photographer information.The photographer information is added to the out-images as additionalinformation. The image processing apparatus 11 then displays a list ofthe detected out-images. Additionally, when displaying the list of theout-images, the image processing apparatus 11 combines the binary imageswith the respective out-images for display according to a user'sinstruction.

If the photographer information that is added to each out-image andserves as additional information is used in such a manner, it ispossible to more effectively display the out-images, for example, todisplay only the out-images of a specific photographer, and also improveconvenience. Additionally, if the encoded binary image as additionalinformation is used, it is possible to display only the out-image ordisplay the out-image with which the binary image is combined, when alist of the out-images is displayed.

Third Embodiment

<Configuration Example of Signal Processing Unit>

Although the case where the encoded binary image and the photographerinformation are generated as additional information has been describedhereinabove, information indicating a gaze area on the out-image may begenerated as additional information. Here, the gaze area refers to anarea on the out-image, at which the user as a photographer gazes whencapturing the out-image.

In the case where information indicating a gaze area is generated asadditional information in such a manner, the signal processing unit 53has a configuration shown in FIG. 13, for example. It should be notedthat in FIG. 13, the same reference numerals are given to portionscorresponding to the portions in FIG. 4 and description thereof will beappropriately omitted.

The signal processing unit 53 shown in FIG. 13 includes a gaze-areadetecting unit 151, an additional processing unit 82, an extraction unit83, a cost calculating unit 152, and a reproduction unit 85. In otherwords, the signal processing unit 53 of FIG. 13 has a configuration inwhich the signal processing unit 53 of FIG. 4 includes the gaze-areadetecting unit 151 and the cost calculating unit 152 instead of thecompression unit 81 and the decoding unit 84.

The gaze-area detecting unit 151 detects a gaze area or an area on theout-image at which the user is gazing, based on an in-image suppliedfrom the in-image capturing unit 22, and supplies a result of thedetection to the additional processing unit 82 as additionalinformation.

The additional processing unit 82 adds information indicating the resultof the detection of gaze area, which is supplied from the gaze-areadetecting unit 151, to the image data of an out-image supplied from theout-image capturing unit 23, the information serving as additionalinformation. The additional processing unit 82 supplies the resultantinformation to the recording unit 54.

The cost calculating unit 152 calculates, as cost, the degree of overlapof gaze areas in the case where a plurality of out-images aresimultaneously or sequentially superimposed and displayed, based on theinformation indicating the result of the detection of gaze area asadditional information supplied from the extraction unit 83. The costcalculating unit 152 supplies the cost to the reproduction unit 85.

The reproduction unit 85 controls the display of the out-images by thedisplay unit 21 based on the cost supplied from the cost calculatingunit 152, and superimposes the plurality of out-images for display suchthat the gaze areas of the respective out-images do not overlap eachother as much as possible.

<Description on Imaging Processing>

Next, with reference to a flowchart of FIG. 14, description will begiven on imaging processing performed when the signal processing unit 53has the configuration shown in FIG. 13.

Since processing of Steps S131 and S132 are similar to the processing ofSteps S11 and S12 of FIG. 5, description thereof will be omitted.

Additionally, in Step S131, at the time when an out-image is captured,an image that is taken in by the out-image capturing unit 23 is suppliedas a preview image of the out-image to the display unit 21 via thereproduction unit 85 of the signal processing unit 53, and thendisplayed. In other words, with the preview image of the out-image beingdisplayed on the display unit 21, the user operates the operation inputunit 51 to give an instruction to capture an out-image.

In Step S133, the gaze-area detecting unit 151 detects a line-of-sightdirection of the user at the time when an out-image is captured, basedon the in-image supplied from the in-image capturing unit 22, to detecta gaze area of the user on the out-image.

For example, the gaze-area detecting unit 151 detects an area of a humanface from the in-image, to determine an eye area on the detected facearea. The gaze-area detecting unit 151 then performs pattern matchingusing the determined eye area and a learning data group prepared inadvance, to obtain a line-of-sight direction of the user. For example,in the pattern matching, a line-of-sight that is associated withlearning data having a highest score in matching with the eye area is aline-of-sight direction to be obtained.

Additionally, the gaze-area detecting unit 151 calculates a relativepositional relationship between an actual position of the face of theuser and the in-image capturing unit 22 and display unit 21, based onthe position and the size of the face of the user on the in-image, anddetects a gaze area based on the positional relationship and theline-of-sight direction. In other words, since the preview image of theout-image is displayed on the display unit 21 when the in-image iscaptured, an area of the out-image, which is displayed on an area of thedisplay unit 21 and at which the user is gazing when the in-image iscaptured, should be an area at which the user is gazing.

It should be noted that the technology to detect a line of sight andspecify a gaze area of a user is described in, for example, thespecification of U.S. Pat. No. 6,246,779.

When detecting a gaze area on the out-image based on the in-image, as aresult of the detection, the gaze-area detecting unit 151 suppliescoordinates of a rectangular area (coordinates of a rectangle) on theout-image to the additional processing unit 82 as additionalinformation, the rectangular area indicating a gaze area, for example.

In Step S134, the additional processing unit 82 adds the additionalinformation, which is supplied from the gaze-area detecting unit 151, tothe image data of the out-image supplied from the out-image capturingunit 23 and supplies the resultant data to the recording unit 54.

In Step S135, the recording unit 54 records the out-image supplied fromthe additional processing unit 82, and the imaging processing isterminated.

As described above, the image processing apparatus 11 obtainsinformation indicating a gaze area of the out-image as additionalinformation, the information being detected based on the in-imagesimultaneously captured with the out-image. The image processingapparatus 11 then adds the additional information to the out-image.

In such a manner, the additional information is added to the out-image,and thus it is possible to display an image more effectively and alsoimprove convenience using the additional information at the time whenthe out-image is reproduced. For example, when a plurality of out-imagesare simultaneously displayed, the plurality of out-images can bedisplayed such that the gaze areas of the respective out-images do notoverlap each other as much as possible. Information that indicates agaze area and is generated as additional information is informationobtained only when the out-image is captured. If such information isused when the out-image is displayed, the out-image can be effectivelypresented.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display theout-images in a slide show. At that time, the user operates theoperation input unit 51, for example, and can thus select whether todisplay out-images one by one or whether to superimpose the plurality ofout-images for simultaneous display.

Hereinafter, with reference to a flowchart of FIG. 15, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to display out-images ina slide show.

Since processing of Steps S161 and S162 are similar to the processing ofSteps S41 and S42 of FIG. 7, description thereof will be omitted.

In Step S161, however, a plurality of out-images such as out-images in afolder specified by the user are read, for example. Additionally, inStep S162, information indicating a gaze area as additional informationread from each of the out-images is supplied from the extraction unit 83to the cost calculating unit 152.

In Step S163, the cost calculating unit 152 calculates the degree ofoverlap of subjects, as cost, based on the information indicating gazeareas as additional information supplied from the extraction unit 83,and supplies the cost to the reproduction unit 85.

Specifically, the cost calculating unit 152 calculates the degree ofimportance of each area of the out-image, with the gaze area being as asaliency (important point). For example, the degree of importance of agaze area is raised and the degree of importance of other areas exceptfor the gaze area is lowered. At that time, the degree of importance maybe raised more at a position closer to the center of the gaze area.

Based on the thus calculated degree of importance of each area of theout-image, the cost calculating unit 152 calculates the cost of thedegree of overlap of subjects for the way to superimpose the pluralityof out-images simultaneously displayed, and supplies the cost to thereproduction unit 85. For example, when the out-images are superimposedin such a manner that an area having a higher degree of importance ishidden by another overlapping out-image, cost is calculated so as to beraised more.

In Step S164, the reproduction unit 85 determines whether to superimposethe plurality of out-images for display or not.

When it is determined in Step S164 that the plurality of out-images aresuperimposed for display, in Step S165, the combining unit 95superimposes and combines the plurality of out-images supplied from theextraction unit 83, based on the cost supplied from the cost calculatingunit 152.

For example, the combining unit 95 combines the plurality of out-imagesby a superimposing way determined by the lowest cost supplied from thecost calculating unit 152. Consequently, the plurality of out-images aresuperimposed such that main subjects of the respective out-images do notoverlap each other as much as possible, that is, the gaze areas are nothidden by other out-images.

The reproduction unit 85 supplies the plurality of combined out-imagesto the display unit 21 and gives an instruction to display the pluralityof out-images in a slide show.

In Step S166, the display unit 21 displays the plurality of combinedout-images, which are supplied from the reproduction unit 85, in a slideshow with the plurality of out-images being simultaneously displayed.The reproduction processing is terminated.

Consequently, the out-images shown in FIG. 16, for example, aredisplayed on the display unit 21.

In the example of FIG. 16, three out-images P11-1 to P11-3 are displayedon the display unit 21. Those out-images partially overlap each other.In particular, in this example, the three out-images are superimposedsuch that main subjects of the respective out-images are not hidden byother out-images.

It should be noted that in a method of superimposing the plurality ofout-images for display, the out-images may be sequentially superimposedfor display one by one, or may be superimposed for simultaneous displayand then followed by a plurality of different out-images that aresimultaneously displayed.

The technology to superimpose and display a plurality of images based ona saliency of image is described in, for example, the specification ofU.S. Pat. No. 7,595,809.

Referring back to the description on the flowchart of FIG. 15, when itis determined in Step S164 that the out-images are not superimposed fordisplay, the reproduction unit 85 sequentially supplies the out-images,which are supplied from the extraction unit 83, to the display unit 21.The processing proceeds to Step S167.

In Step S167, the display unit 21 sequentially display the out-imagessupplied from the reproduction unit 85, to display the out-images in aslide show. The reproduction processing is terminated.

As described above, the image processing apparatus 11 superimposes theplurality of out-images for display such that the different out-imagesdo not overlap each other at the respective gaze areas as much aspossible, based on the information indicating the gaze areas, theinformation being added as additional information to the out-images. Insuch a manner, the additional information that can be obtained only whenthe out-images are captured is used, and thus the out-images can bedisplayed more effectively.

Hereinabove, the example in which the information indicating gaze areasas additional information is used for superimposing the out-images fordisplay has been described, but the information indicating gaze areasmay be used for another method of displaying out-images. For example, inthe case where the out-images are sequentially displayed one by one,based on the information indicating gaze areas, the gaze area of theout-image may be enlarged for display (zoomed).

Fourth Embodiment

<Configuration Example of Signal Processing Unit>

A part of the in-image may be obtained as additional information, andthe image serving as the additional information may be combined with anout-image at the time when the out-image is reproduced.

Specifically, as shown in FIG. 17, for example, it is assumed that anout-image OT11 having a subject of a landscape and an in-image IN11having a subject of a user are captured.

In this case, viewing-angle conversion processing is performed on theout-image OT11, and the resultant image T11 is obtained as a finalout-image. In this example, the image T11 is an image obtained byconverting an aspect ratio of the out-image OT11. In the image T11, allthe subjects on the out-image OT11 appear. In other words, the image T11is an image obtained by contracting the out-image OT11 in a horizontaldirection of the figure.

Additionally, processing such as face recognition is performed on thein-image IN11, an area of a main subject is detected from the in-imageIN11, and an area including the area of the main subject of the in-imageIN11 is cut out, to obtain an image C11 as additional information. Inparticular, in the case where the main subject is a human face, whenimage transformation processing is performed on the area of the mainsubject, the human face is distorted. Here, the area of the main subjectis cut out, and thus an image without distortion is obtained.

It should be noted that a ratio of a length in a vertical direction to alength in a horizontal direction of the image T11 in the figure isassumed to be specified in advance as an out-side aspect parameter.Similarly, a ratio of a length in a vertical direction to a length in ahorizontal direction of the image C11 in the figure is assumed to bespecified in advance as an in-side aspect parameter.

Additionally, those out-side aspect parameter and in-side aspectparameter are determined such that an aspect ratio of an image, which isobtained when the image T11 and the image C11 are arranged in thehorizontal direction in the figure to become one image, becomes the sameas the aspect ratio of the original out-image OT11.

When the image C11 is obtained in such a manner, the image C11 is addedto the image data of the image T11 as additional information.

At the time when the image T11 is reproduced, the user can selectwhether to reproduce the image T11 as it is or whether to combine theimage C11 as additional information with the image T11 for display.

For example, in the case where an instruction is given to display animage in which the image C11 is combined with the image T11, the imageprocessing apparatus 11 arranges the image T11 and the image C11 in thehorizontal direction in the figure and combines those images, togenerate a composite image P21. The image processing apparatus 11 thendisplays the obtained composite image P21.

For example, in the case where the in-image and the out-image arepartially cut out to be coupled to each other, there is a possibilitythat main subjects or landscapes are deleted. In contrast to this, thecomposite image P21 is an image showing a state of a photographer whenthe out-image OT11 is captured, while leaving all the content of theoriginal out-image OT11, and thus it is possible to more effectivelydisplay images.

Additionally, in the case where the in-image and the out-image aresimply arranged to be coupled to each other, the aspect ratios of theimages are changed, and due to the difference in aspect ratio from thedisplay unit 21, for example, a trouble occurs in which the imagesshould be displayed in small size. In contrast to this, the compositeimage P21 has the same aspect ratio as the original out-image OT11, andthus there is no trouble in display.

In the case where a part of the in-image is cut out to be obtained asadditional information in such a manner, the signal processing unit 53has a configuration shown in FIG. 18, for example. It should be notedthat in FIG. 18, the same reference numerals are given to portionscorresponding to the portions in FIG. 4 and description thereof will beappropriately omitted.

The signal processing unit 53 shown in FIG. 18 includes a viewing-angleconverting unit 181, an additional processing unit 82, a main-areadetecting unit 182, a crop processing unit 183, an extraction unit 83,and a reproduction unit 85.

The viewing-angle converting unit 181 performs viewing-angle conversionprocessing on an out-image supplied from the out-image capturing unit 23such that the out-image obtains a target aspect ratio based on aspecified out-side aspect parameter, and supplies the resultant image tothe additional processing unit 82.

The main-area detecting unit 182 detects an area of a main subject(hereinafter, referred to as main area) from an in-image supplied fromthe in-image capturing unit 22 and supplies a result of the detectionand the in-image to the crop processing unit 183.

Based on the result of the detection on the main area, which is suppliedfrom the main-area detecting unit 182, and the specified in-side aspectparameter, the crop processing unit 183 cuts out an area including themain area of the in-image to obtain a cropped image, and supplies thecropped image to the additional processing unit 82. In other words, acropped image obtained by the crop processing performed on the in-imageis supplied to the additional processing unit 82 as additionalinformation.

The additional processing unit 82 adds the cropped image to theout-image supplied from the viewing-angle converting unit 181, thecropped image being supplied from the crop processing unit 183 asadditional information. The additional processing unit 82 supplies theresultant image to the recording unit 54.

The extraction unit 83 extracts the additional information from theout-image read from the recording unit 54 and supplies the out-image andthe additional information to the reproduction unit 85.

<Description on Imaging Processing>

Next, with reference to the flowchart of FIG. 19, description will begiven on imaging processing performed by the image processing apparatus11 when the signal processing unit 53 has the configuration shown inFIG. 18.

Since processing of Steps S191 and S192 are similar to the processing ofSteps S11 and S12 of FIG. 5, description thereof will be omitted.

In Step S193, the viewing-angle converting unit 181 performsviewing-angle conversion processing on the out-image supplied from theout-image capturing unit 23, based on a specified out-side aspectparameter, and supplies the resultant image to the additional processingunit 82.

For example, the viewing-angle converting unit 181 converts the viewingangle of the out-image by seam carving processing. In the seam carvingprocessing, a path in the vertical direction in which the cost becomesthe lowest is calculated until the aspect ratio of the out-image reachesan aspect ratio indicated by the out-side aspect parameter, andprocessing of thinning out pixels on the path is repeated. In otherwords, the thinning-out of the pixels is performed such that a change ofa pattern from the original image becomes the least.

Consequently, the image T11 shown in FIG. 17, for example, is obtainedas a final out-image and supplied to the additional processing unit 82.

It should be noted that the example of the seam carving processing hasbeen described as the viewing-angle conversion processing in which theviewing angle of the out-image is converted, but processing ofcontracting an image may be performed as the viewing-angle conversionprocessing.

In Step S194, the main-area detecting unit 182 detects a main area fromthe in-image supplied from the in-image capturing unit 22, and suppliesa result of the detection and the in-image to the crop processing unit183. For example, face recognition processing is performed on thein-image, and an area of a human face in the in-image is obtained as amain area.

In Step S195, the crop processing unit 183 performs crop processing onthe in-image supplied from the main-area detecting unit 182, based onthe result of the detection on the main area supplied from the main-areadetecting unit 182 and on a specified in-side aspect parameter.

In other words, the crop processing unit 183 cuts out an area includingthe main area in the in-image and having an aspect ratio indicated bythe in-side aspect parameter to obtain a cropped image, and supplies thecropped image to the additional processing unit 82. Consequently, theimage C11 shown in FIG. 17, for example, is obtained as a cropped image.The cropped image thus obtained is the additional information of theout-image.

In Step S196, the additional processing unit 82 adds the cropped image,which is supplied from the crop processing unit 183 and serves asadditional information, to the image data of the out-image supplied fromthe viewing-angle converting unit 181, and supplies the resultant imageto the recording unit 54.

In Step S197, the recording unit 54 records the out-image supplied fromthe additional processing unit 82, and the imaging processing isterminated.

As described above, the image processing apparatus 11 obtains a croppedimage as additional information, the cropped image being obtained bycutting out an area including the main area of the in-image, and addsthe cropped image to the out-image. In such a manner, the cropped imageas additional information is embedded in the out-image, and thus it ispossible to display only an out-image or combine the cropped image withthe out-image for display. Consequently, it is possible to moreeffectively display the out-image and also improve convenience.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display theout-image. At that time, the user operates the operation input unit 51,for example, and can thus select whether to display only the out-imageor whether to combine the cropped image as additional information withthe out-images for display.

Hereinafter, with reference to a flowchart of FIG. 20, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to reproduce theout-image.

In Step S221, the extraction unit 83 reads the out-image from therecording unit 54, the out-image being specified by the user accordingto an instruction of the control unit 52.

In Step S222, the extraction unit 83 reads the cropped image asadditional information from the image data of the out-image and suppliesthe cropped image and the out-image to the reproduction unit 85.

In Step S223, the reproduction unit 85 determines whether to combine thecropped image with the out-image or not.

When it is determined in Step S223 that the cropped image is combined,in Step S224, the combining unit 95 combines the cropped image with theout-image supplied from the extraction unit 83. Specifically, thecombining unit 95 arranges the out-image and the cropped image andperforms weighted addition processing of the out-image and the croppedimage in areas adjacent to the boundary of those images, to generate onecomposite image. Consequently, the composite image P21 shown in FIG. 17,for example, is obtained.

Here, a weight used in the weighted addition processing is determinedsuch that a position closer to the out-image side has a greater weightof the out-image, for example. More specifically, in the case where theweighted addition processing is performed, the size of the out-image andthat of the cropped image are increased by a proportion corresponding toan area required for the weighted addition processing, and a compositeimage eventually obtained has the same aspect ratio as the originalout-image.

When the composite image is obtained in such a manner, the reproductionunit 85 supplies the composite image to the display unit 21.

In Step S225, the display unit 21 displays the composite image suppliedfrom the reproduction unit 85, and the reproduction processing isterminated.

Additionally, when it is determined in Step S223 that the cropped imageis not combined, the reproduction unit 85 supplies the out-image as itis to the display unit 21, the out-image being supplied from theextraction unit 83, and the processing proceeds to Step S226.

In Step S226, the display unit 21 displays the out-image supplied fromthe reproduction unit 85, and the reproduction processing is terminated.

As described above, the image processing apparatus 11 displays theout-image as it is or combines the cropped image with the out-image fordisplay, the cropped image being added as additional information,according to the instruction of the user.

If the additional information added to the out-image is used in such amanner, it is possible to display a desired image out of the out-imageand the composite image. In other words, it is possible to moreeffectively display the out-image and provide more enjoyment to the userwhen viewing the out-image.

Fifth Embodiment

<Configuration Example of Signal Processing Unit>

In the second embodiment described above, the example in which thephotographer information as additional information is added to theout-image has been described, but a recording area for the out-image maybe determined according to the photographer information as additionalinformation.

For example, as shown in FIG. 21, it is assumed that four out-images P41to P44 are captured, and photographer information are obtained asadditional information of those out-images. In this example, when theout-images P41 to P44 are reproduced, binary images IP41 to IP44 arecombined with the out-images P41 to P44 for display as necessary.

In the case where the out-images P41 to P44 are obtained in such amanner, those out-images are recorded in folders on a photographerbasis, the photographers being indicated by the pieces of photographerinformation of the out-images.

For example, as shown on the right side in the figure, the recordingunit 54 is provided in advance with a folder FD11 of a photographer Aand a folder FD12 of a photographer B.

It is assumed that the photographer information added as additionalinformation to the out-images P41 and P44 is information indicating thephotographer A, and the photographer information added as additionalinformation to the out-images P42 and P43 is information indicating thephotographer B. In such a case, the out-images P41 and P44 are recordedin the folder FD11 and the out-images P42 and P43 are recorded in thefolder FD12.

In such a manner, in the case where the out-images are recorded in thefolders on a photographer basis as recording areas determined by thephotographer information, the signal processing unit 53 has aconfiguration shown in FIG. 22, for example. In FIG. 22, the samereference numerals are given to portions corresponding to the portionsin FIG. 8 and description thereof will be appropriately omitted.

The signal processing unit 53 shown in FIG. 22 includes a compressionunit 81, an additional processing unit 82, an extraction unit 83, adecoding unit 84, a reproduction unit 85, a face recognition unit 121,and a classification unit 211. In other words, the signal processingunit 53 of FIG. 22 has a configuration in which the signal processingunit 53 of FIG. 8 further includes the classification unit 211.

The classification unit 211 supplies the out-image from the additionalprocessing unit 82 to the recording unit 54 for recording so as to bestored in a folder determined by the photographer information added asadditional information to the out-image, out of the folders of therecording unit 54.

<Description on Imaging Processing>

Next, with reference to a flowchart of FIG. 23, description will begiven on imaging processing performed when the signal processing unit 53has the configuration shown in FIG. 22.

Since processing of Steps S251 to S257 are similar to the processing ofSteps S71 to S77 of FIG. 9, description thereof will be omitted. In StepS257, however, the image data of the out-image, to which thephotographer information and the encoded binary image are added asadditional information, is supplied from the additional processing unit82 to the classification unit 211.

In Step S258, the classification unit 211 records the out-image in therecording unit 54 according to the additional information of theout-image supplied from the additional processing unit 82, and theimaging processing is terminated.

Specifically, the photographer information is associated with each ofthe folders provided in the recording unit 54, for example. Theclassification unit 211 identifies a folder with which the samephotographer information as the photographer information added to theout-image is associated. The classification unit 211 supplies theout-image to the recording unit 54 so as to be stored in the identifiedfolder and then recorded therein. In other words, the out-image isrecorded in the recording area of the recording unit 54, the recordingarea being assumed as the area of the identified folder.

As described above, the image processing apparatus 11 adds the binaryimage and the photographer information, which are obtained as additionalinformation from the in-image, to the out-image, and records theout-image in the folder determined by the photographer information.

Consequently, it is possible to more effectively display the out-imageand also improve convenience. For example, if the photographerinformation generated as additional information is used, it is possibleto effectively display the out-image, for example, to display a list ofout-images captured by a specific photographer. In addition, since theout-images are recorded in the folders on a photographer basis accordingto the photographer information at the time when the out-images arerecorded, the out-images of a specific photographer can be quicklyidentified.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display a list ofout-images captured by a specific user. At that time, the user operatesthe operation input unit 51, for example, and can thus select whether todisplay only out-images or whether to combine images, which are obtainedfrom additional information, with the out-images for display.

Hereinafter, with reference to a flowchart of FIG. 24, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to display a list ofout-images.

In Step S281, the extraction unit 83 reads all out-images recorded in aspecific folder of the recording unit 54. For example, the extractionunit 83 reads out-images that are stored in a folder of any user(photographer) specified by the user operating the operation input unit51.

Additionally, for example, the extraction unit 83 may read an out-imagethat is stored in a folder of a user (photographer) logging in the imageprocessing apparatus 11. Further, it may be possible to capture anin-image by the in-image capturing unit 22, perform face recognition bythe face recognition unit 121, and read an out-image stored in a folderof a user identified as a result of the face recognition, that is, auser operating the image processing apparatus 11.

After the out-images are read in such a manner, processing of Steps S282to S288 are performed, and the reproduction processing is terminated.Since those processing are similar to the processing of Steps S103 toS109 in FIG. 10, description thereof will be omitted.

As described above, the image processing apparatus 11 reads theout-images from the folders that are provided on a photographer basis,the photographers being indicated by the photographer information addedto the out-images as additional information, to display a list of theout-images. Additionally, the image processing apparatus 11 combines thebinary images with the out-images for display according to aninstruction of the user, when the out-images are displayed in a list.

If the photographer information added to the out-image as additionalinformation is used in such a manner, it is possible to more effectivelydisplay the out-images, for example, to display only the out-images of aspecific photographer. Additionally, if the out-image is stored in afolder that is determined by the photographer information as additionalinformation, an out-image of a specific photographer can be identifiedmore easily and quickly, which improves convenience.

Sixth Embodiment

<Configuration Example of Signal Processing Unit>

A smile of a user operating the image processing apparatus 11 may bedetected to capture an out-image and an in-image at a timing when thesmile is detected, and the degree of smile of the user may be added tothe out-image as additional information.

In such a case, the signal processing unit 53 has, for example, aconfiguration shown in FIG. 25. In FIG. 25, the same reference numeralsare given to portions corresponding to the portions in FIG. 8 anddescription thereof will be appropriately omitted.

The signal processing unit 53 shown in FIG. 25 includes a compressionunit 81, an additional processing unit 82, an extraction unit 83, adecoding unit 84, a reproduction unit 85, a face recognition unit 121,and a smile detecting unit 241. In other words, the signal processingunit 53 of FIG. 25 has a configuration in which the signal processingunit 53 of FIG. 8 further includes the smile detecting unit 241.

The smile detecting unit 241 detects a smile of a user who operates theimage processing apparatus 11, that is, a user as a photographer of theout-image, from the in-image supplied from the in-image capturing unit22. The smile detecting unit 241 supplies information to the controlunit 52 according to a result of the detection, the informationindicating that an imaging timing has come. The control unit 52 controlsthe out-image capturing unit 23 and the in-image capturing unit 22 tocapture an out-image and an in-image according to the informationindicating that an imaging timing has come, the information beingsupplied from the smile detecting unit 241.

Additionally, the smile detecting unit 241 generates the degree of smileof the user, the degree of smile being detected from the in-image andserving as additional information, and supplies the additionalinformation to the additional processing unit 82.

<Description on Imaging Processing>

Next, with reference to a flowchart of FIG. 26, description will begiven on imaging processing performed when the signal processing unit 53has the configuration shown in FIG. 25. In this case, the user operatesthe operation input unit 51, and when the user gives an instruction tocapture an out-image, imaging processing is started.

In Step S311, the in-image capturing unit 22 captures an in-imageaccording to an instruction of the control unit 52 and supplies thein-image to the compression unit 81, the face recognition unit 121, andthe smile detecting unit 241.

In Step S312, the smile detecting unit 241 detects a smile of the userfrom the in-image supplied from the in-image capturing unit 22.

For example, the smile detecting unit 241 holds learning data of a largenumber of smiles determined in advance. Here, the learning data refersto, for example, image data of human faces having facial expressionssuch as a faint smile and a loud laughter, regardless of specificsmiles.

The smile detecting unit 241 compares a feature amount obtained fromeach piece of the learning data with a feature amount obtained from thein-image to calculate a score value, and thus calculates the degree ofsmile of the user on the in-image. At that time, the degree ofsimilarity to the in-image is considered as the degree of smile of theuser on the in-image. Additionally, for example, in the case where thedegree of smile has a specific value or more, it is considered that asmile is detected from the in-image.

It should be noted that a method of detecting a smile may be any methodas long as a smile can be detected from the in-image. For example, thesmile detecting unit 241 may hold learning data on a degree-of-smilebasis, for a plurality of degrees of smile determined in advance. Here,the learning data refers to, for example, image data of an average humansmile, which is a specific degree of smile, that is, image data of anaverage human face having specific facial expressions such as a faintsmile and a loud laughter.

The smile detecting unit 241 compares a feature amount obtained fromeach piece of the learning data with a feature amount obtained from thein-image and identifies learning data that is the most similar to thein-image from the pieces of the learning data, to calculate the degreeof smile of the user on the in-image. At that time, in the case wherethere is learning data in which the degree of similarity to the in-imageis a specific value or more, it is considered that a smile is detectedfrom the in-image and that the degree of smile in the learning datahaving the highest degree of similarity is considered as the degree ofsmile of the user on the in-image.

In Step S313, the smile detecting unit 241 determines whether a smile isdetected from the in-image or not.

When it is determined in Step S313 that a smile is not detected, theprocessing returns to Step S311, and the processing described above isrepeated.

In contrast to this, when it is determined in Step S313 that a smile isdetected, the smile detecting unit 241 supplies information to thecontrol unit 52, the information indicating that an imaging timing hascome, and the processing proceeds to Step S314. Additionally, the smiledetecting unit 241 supplies the degree of smile of the user on thein-image to the additional processing unit 82, the degree of smile beingobtained by smile detection and serving as additional information.

It should be noted that here, the degree of smile as additionalinformation of an out-image is information indicating a smile (facialexpression) of a photographer who captures the out-image. Additionally,in calculation of the degree of smile in the smile detecting unit 241,the mouth shape, the size of a teeth area, and the like in the user'sface that are detected from the in-image may be used.

After a smile is detected from the in-image, processing of Steps S314 toS319 are performed. Since those processing are similar to the processingof Steps S71 to S76 in FIG. 9, description thereof will be omitted. InSteps S314 and S315, however, the control unit 52 causes the out-imagecapturing unit 23 and the in-image capturing unit 22 to capture anout-image and an in-image based on information indicating that animaging timing has come, the information being supplied from the smiledetecting unit 241. In other words, an out-image and an in-image arecaptured at the timing when the smile of the photographer is detected.

In Step S320, the additional processing unit 82 obtains, as additionalinformation, the photographer information from the face recognition unit121, the encoded binary image from the compression unit 81, and thedegree of smile from the smile detecting unit 241, and adds theadditional information to the image data of the out-image from theout-image capturing unit 23. The additional processing unit 82 thensupplies the image data of the out-image, to which the additionalinformation is added, to the recording unit 54.

In Step S321, the recording unit 54 records the out-image supplied fromthe additional processing unit 82, and the imaging processing isterminated.

As described above, the image processing apparatus 11 adds the binaryimage, the photographer information, and the degree of smile, which areobtained as additional information from the in-image, to the out-image.

Consequently, it is possible to more effectively display the out-imageand also improve convenience. For example, if the degree of smile isused as additional information, it is possible to selectively display anout-image in which a photographer has a specific facial expression, thatis, a specific degree of smile, at the time when the image is captured.Additionally, if the degree of smile is obtained as additionalinformation, an out-image in a specific degree of smile can be detectedmore easily and quickly, which improves convenience.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display out-imageswith a specific degree of smile in a slide show or in a list. At thattime, the user operates the operation input unit 51, for example, andcan thus select whether to display only out-images or whether to combineimages, which are obtained from additional information, with theout-images for display.

Hereinafter, with reference to a flowchart of FIG. 27, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to display out-images ina slide show.

In Step S351, the extraction unit 83 reads all out-images recorded inthe recording unit 54.

In Step S352, the extraction unit 83 reads the degree of smile asadditional information, which is added to each of the read out-images,and detects out-images with a specific degree of smile. For example, aspecific degree of smile is specified in advance by the user.

After out-images with a specific degree of smile are detected, theprocessing of Steps S353 to S357 are performed. Since those processingare similar to the processing of Steps S103 to S107 in FIG. 10,description thereof will be omitted. In Step S353, however, theadditional information is read from each of the out-images with thespecific degree of smile, and those out-images are supplied to thereproduction unit 85. Additionally, the reproduction unit 85sequentially supplies the out-images, with which the binary images arecombined, to the display unit 21.

In Step S358, the display unit 21 sequentially displays the out-images,with which the binary images are combined and which are supplied fromthe reproduction unit 85, and the reproduction processing is terminated.

In contrast to this, when it is determined in Step S356 that the binaryimage are not reproduced, the reproduction unit 85 supplies theout-images, which are supplied from the extraction unit 83, to thedisplay unit 21 as they are, and the processing proceeds to Step S359.

In Step S359, the display unit 21 sequentially displays the out-imagessupplied from the reproduction unit 85, and the reproduction processingis terminated.

As described above, the image processing apparatus 11 detects anddisplays the out-images with a specific degree of smile, using thedegree of smile added to the out-images as additional information.Additionally, when the out-images are displayed, the image processingapparatus 11 combines the binary image with each out-image for display,according to a user's instruction.

If the degree of smile as additional information added to the out-imageis used in such a manner, it is possible to more effectively display theout-image, for example, to display only the out-image captured when thephotographer has a specific facial expression.

Seventh Embodiment

<Configuration Example of Signal Processing Unit>

Although the example in which the degree of smile is added as additionalinformation of the out-image has been described hereinabove, a recordingarea for the out-image may be determined according to the degree ofsmile as additional information.

For example, as shown in FIG. 28, it is assumed that three out-imagesP51 to P53 are captured, and the degrees of smile are obtained asadditional information of those out-images. In this example, when theout-images P51 to P53 are reproduced, binary images IP51 to IP53 arecombined with the out-images P51 to P53 for display as necessary.

In the case where the out-images P51 to P53 are obtained in such amanner, those out-images are recorded in folders on adegree-of-photographer's smile basis.

For example, as shown on the right side in the figure, the recordingunit 54 is provided in advance with a folder FD21 of the degree of smileA and a folder FD22 of the degree of smile B.

It is assumed that the degree of smile added as additional informationto the out-images P51 and P53 is information indicating the degree ofsmile B, and the degree of smile added as additional information to theout-image P52 is information indicating the degree of smile A. In such acase, the out-image P52 is recorded in the folder FD21, and theout-images P51 and P53 are recorded in the folder FD22.

In such a manner, in the case where the out-images are recorded in thefolders on a degree-of-smile basis as recording areas determined by thedegrees of smile of the photographers, the signal processing unit 53 hasa configuration shown in FIG. 29, for example. In FIG. 29, the samereference numerals are given to portions corresponding to the portionsin FIG. 25 and description thereof will be appropriately omitted.

The signal processing unit 53 shown in FIG. 29 includes a compressionunit 81, an additional processing unit 82, an extraction unit 83, adecoding unit 84, a reproduction unit 85, a face recognition unit 121, asmile detecting unit 241, and a classification unit 271. In other words,the signal processing unit 53 of FIG. 29 has a configuration in whichthe signal processing unit 53 of FIG. 25 further includes theclassification unit 271.

The classification unit 271 supplies the out-image from the additionalprocessing unit 82 to the recording unit 54 for recording so as to bestored in a folder determined by the degree of smile added as additionalinformation to the out-image, out of the folders of the recording unit54.

<Description on Imaging Processing>

Next, with reference to a flowchart of FIG. 30, description will begiven on imaging processing performed when the signal processing unit 53has the configuration shown in FIG. 29.

Since processing of Steps S381 to S390 are similar to the processing ofSteps S311 to S320 of FIG. 26, description thereof will be omitted. InStep S390, however, the image data of the out-image, to which thephotographer information, the encoded binary image, and the degree ofsmile are added as additional information, is supplied from theadditional processing unit 82 to the classification unit 271.

In Step S391, the classification unit 271 records the out-image in therecording unit 54 according to the additional information of theout-image supplied from the additional processing unit 82, and theimaging processing is terminated.

Specifically, the degree of smile is associated with each of the foldersprovided in the recording unit 54, for example. The classification unit271 identifies a folder with which the same degree of smile as thedegree of smile added to the out-image is associated. The classificationunit 271 then supplies the out-image to the recording unit 54 so as tobe stored in the identified folder and then recorded therein. In otherwords, the out-image is recorded in the recording area of the recordingunit 54, the recording area being assumed as the area of the identifiedfolder.

As described above, the image processing apparatus 11 adds the binaryimage, the photographer information, and the degree of smile, which areobtained as additional information from the in-image, to the out-image,and records the out-image in the folder determined by the degree ofsmile.

Consequently, it is possible to more effectively display the out-imageand also improve convenience. For example, if the degree of smilegenerated as additional information is used, it is possible toeffectively display the out-image, for example, to display the out-imagecaptured in a specific facial expression. In addition, since theout-images are recorded in folders on a photographer's facialexpressions basis according to the degree of smile at the time when theout-images are recorded, the out-images of a specific degree of smilecan be quickly identified, which improves convenience.

It should be noted that the example in which the folders are provided ona degree-of-smile basis regardless of the photographers has beendescribed here, but the folders may be provided on a degree-of-smilebasis for each photographer. In such a case, the out-images are storedin the folders determined by the photographer information and the degreeof smile of the out-images.

<Description on Reproduction Processing>

Additionally, when the out-image is recorded in the recording unit 54,the user can operate the operation input unit 51 to display theout-image captured by a user having a specific facial expression ordisplay out-images in a list. At that time, the user operates theoperation input unit 51, for example, and can thus select whether todisplay only out-images or whether to combine images, which are obtainedfrom the additional information, with the out-images for display.

Hereinafter, with reference to a flowchart of FIG. 31, description willbe given on reproduction processing performed by the image processingapparatus 11 when the user gives an instruction to display an out-image.

In Step S431, the extraction unit 83 reads all out-images recorded in aspecific folder of the recording unit 54. For example, the extractionunit 83 reads out-images that are stored in a folder of any degree ofsmile specified by the user operating the operation input unit 51.

After the out-images are read in such a manner, the processing of StepsS432 and S438 are performed, and the reproduction processing isterminated. Since those processing are similar to the processing ofSteps S353 to S359 in FIG. 27, description thereof will be omitted.

As described above, the image processing apparatus 11 reads theout-images from the folders that are provided on a photographer's facialexpressions basis indicated by the degree of smile, the degree of smilebeing added as additional information to the out-images, to display theout-images. Additionally, the image processing apparatus 11 combines thebinary images with the out-images for display according to aninstruction of the user, when the out-images are displayed.

In such a manner, if the degree of smile as additional information addedto the out-image is used, it is possible to more effectively display theout-image, for example, to display only the out-image captured when thephotographer has a specific facial expression. Additionally, if theout-image is stored in a folder that is determined by the degree ofsmile as additional information, an out-image of a specific degree ofsmile can be identified more easily and quickly, which improvesconvenience.

Eighth Embodiment

<Description on Imaging Processing>

In the seventh embodiment described above, the example in which anout-image is captured when a smile is detected from the in-image hasbeen described, but an out-image and an in-image may be capturedaccording to a user's instruction and the out-image may be recorded in afolder corresponding to the degree of smile.

In such a case, the signal processing unit 53 has a configuration shownin FIG. 29, for example. In this example, however, informationindicating that an imaging timing has come is not provided from thesmile detecting unit 241 to the control unit 52. The control unit 52instructs the out-image capturing unit 23 and the in-image capturingunit 22 to capture an out-image and an in-image according to anoperation on the operation input unit 51.

Next, with reference to a flowchart of FIG. 32, description will begiven on imaging processing performed by the image processing apparatus11 when an out-image and an in-image are captured according to a user'sinstruction and the out-image is recorded according to the degree ofsmile.

Since processing of Steps S461 to S466 are similar to the processing ofSteps S251 to S256 of FIG. 23, description thereof will be omitted. Inother words, in those processing, an out-image and an in-image arecaptured according to a user's instruction, and an encoded binary imageand photographer information are generated as additional information ofthe out-image.

In Step S467, the smile detecting unit 241 detects a smile of a userfrom the in-image supplied from the in-image capturing unit 22, andsupplies the degree of smile of the user on the in-image thus obtained,as additional information, to the additional processing unit 82. Forexample, in Step S467, the processing similar to Step S312 of FIG. 26 isperformed.

In Step S468, the additional processing unit 82 obtains, as additionalinformation, photographer information from the face recognition unit121, an encoded binary image from the compression unit 81, and thedegree of smile from the smile detecting unit 241, and adds theadditional information to the image data of the out-image from theout-image capturing unit 23. The additional processing unit 82 thensupplies the image data of the out-image, to which the additionalinformation is added, to the classification unit 271.

In Step S469, the classification unit 271 records the out-image in therecording unit 54 according to the additional information of theout-image supplied from the additional processing unit 82, and theimaging processing is terminated.

Specifically, the classification unit 271 identifies a folder with whichthe same degree of smile as the degree of smile added to the out-imageis associated. The classification unit 271 then supplies the out-imageto the recording unit 54 so as to be stored in the folder and recordedtherein. In other words, in Step S469, the processing similar to StepS391 of FIG. 30 is performed.

As described above, the image processing apparatus 11 adds the binaryimage, the photographer information, and the degree of smile, which areobtained as additional information from the in-image, to the out-image,and records the out-image in the folder determined by the degree ofsmile.

Consequently, it is possible to more effectively display the out-imageand also improve convenience. For example, if the degree of smilegenerated as additional information is used, it is possible toeffectively display the out-image, for example, to display the out-imagecaptured in a specific facial expression. In addition, since theout-images are recorded in folders on a photographer's facialexpressions basis according to the degree of smile at the time when theout-images are recorded, the out-images of a specific degree of smilecan be quickly identified.

Additionally, when the out-image is recorded in the recording unit 54,in the image processing apparatus 11, an out-image captured in aspecific facial expression is displayed according to the operation tothe operation input unit 51 by the user. In such a case, in the imageprocessing apparatus 11, the reproduction processing described withreference to FIG. 31 is performed. In other words, an out-image storedin a folder with the degree of smile specified by the user is displayed.

The above-described series of processing may be performed by hardware ormay be performed by software. In the case where the series of processingis performed by software, a program of the software is installed in acomputer. Here, examples of the computer include a computer incorporatedin dedicated hardware or a general-purpose personal computer capable ofexecuting various functions by installing various programs.

FIG. 33 is a block diagram showing a configuration example of hardwareof a computer executing the above-described series of processing by aprogram.

In the computer, a central processing unit (CPU) 501, a read-only memory(ROM) 502, and a random access memory (RAM) 503 are connected to oneanother by a bus 504.

Further, an input/output interface 505 is connected to the bus 504. Aninput unit 506, an output unit 507, a recording unit 508, acommunication unit 509, and a drive 510 are connected to theinput/output interface 505.

The input unit 506 is configured by a keyboard, a mouse, a microphone,an imaging element, or the like. The output unit 507 is configured by adisplay, a speaker, or the like. The recording unit 508 is configured bya hard disk, a non-volatile memory, or the like. The communication unit509 is configured by a network interface or the like. The drive 510drives a removable medium 511 such as a magnetic disc, an optical disc,a magneto-optical disc, or a semiconductor memory.

In the computer having the above-described configuration, the CPU 501performs the above-described series of processing by loading, forexample, the program recorded on the recording unit 508 to the RAM 503via the input/output interface 505 and the bus 504 and executing theprogram.

For example, the program executed by the computer (the CPU 501) can berecorded on the removable medium 511 as a package medium or the like,and shared. The program can be provided via a wired or wirelesstransmission medium such as a local area network, the Internet, ordigital satellite broadcast.

In the computer, the program can be installed on the recording unit 508via the input/output interface 505 by mounting the removable medium 511on the drive 510. The program can be received by the communication unit509 via a wired or wireless transmission medium and can be installed onthe recording unit 508. In addition thereto, the program can beinstalled in advance in the ROM 502 or the recording unit 508.

The program executed by the computer may be a program processedchronologically in an order described in this specification or may be aprogram processed in parallel or at a necessary timing such as a calledtime.

Embodiments of the present technology are not limited to theabove-described embodiments, but may be modified in various ways withinthe scope of the present technology without departing from the gist ofthe present technology.

For example, the present technology may have a configuration of a cloudcomputing configuration in which one function is distributed andprocessed collectively by a plurality of devices via a network.

Additionally, the respective steps described in the above-describedflowcharts may be executed by one device and may be also distributed andexecuted by a plurality of devices.

Further, when a plurality of processing are included in one step, theplurality of processing included in the one step may be executed by onedevice or may be distributed and executed by a plurality of devices.

Furthermore, the present technology may be configured as follows.

[1] An image processing apparatus, including:

a first imaging unit that images a first direction;

a second imaging unit that images a second direction, the seconddirection being different from the first direction;

an additional-information generating unit that generates additionalinformation based on a first image, the first image being obtained inimaging by the first imaging unit; and

an additional processing unit that adds the additional information toimage data of a second image, the second image being obtained in imagingby the second imaging unit.

[2] The image processing apparatus according to [1], in which

the additional-information generating unit compresses an informationamount of the first image, to generate the additional information.

[3] The image processing apparatus according to [1] or [2], in which

the additional-information generating unit compresses the size of thefirst image, to generate the additional information.

[4] The image processing apparatus according to any one of [1] to [3],in which

the first imaging unit images a direction of a user operating the imageprocessing apparatus, the direction serving as the first direction, and

the image processing apparatus further includes a combining unit thatcombines an image obtained from the additional information with thesecond image.

[5] The image processing apparatus according to [1], in which

the first imaging unit images a direction of a user operating the imageprocessing apparatus, the direction serving as the first direction, and

the additional-information generating unit performs face recognitionprocessing on the first image, to generate photographer information ofthe second image as the additional information.

[6] The image processing apparatus according to [5], further including:

an extraction unit that detects a second image from the plurality ofsecond images based on the additional information, the second imagebeing captured by a specific photographer; and

a reproduction unit that displays only the second image detected by theextraction unit.

[7] The image processing apparatus according to [5], further including aclassification unit that records the second image in a recording areadetermined by the photographer information, the photographer informationserving as the additional information.

[8] The image processing apparatus according to [1], further including adisplay unit that is provided to a surface facing a user operating theimage processing apparatus, the first imaging unit being provided to thesurface, and displays an image taken in by the second imaging unit at atime when the second image is captured, in which

the additional-information generating unit detects a line-of-sightdirection of the user based on the first image, to generate informationindicating a gaze area at which the user gazes on the second image, theinformation serving as the additional information.

[9] The image processing apparatus according to [8], further including areproduction unit that controls display of the second image based on theadditional information.

[10] The image processing apparatus according to [9], in which

the reproduction unit superimposes and displays the plurality of secondimages based on the additional information.

[11] The image processing apparatus according to [1], further includinga viewing-angle converting unit that performs viewing-angle conversionprocessing on the second image, in which

the additional-information generating unit obtains, as the additionalinformation, a cropped image obtained by cutting out an area including amain subject on the first image, and

the additional processing unit adds the additional information to theimage data of the second image on which the viewing-angle conversionprocessing is performed.

[12] The image processing apparatus according to [11], further includinga combining unit that arranges and combines the cropped image serving asthe additional information with the second image on which theviewing-angle conversion processing is performed, to generate acomposite image with an aspect ratio that is the same as an aspect ratioof the second image captured by the second imaging unit.

[13] The image processing apparatus according to [1], in which

the first imaging unit images a direction of a user operating the imageprocessing apparatus, the direction serving as the first direction,

the additional-information generating unit detects a smile of the userbased on the first image and generates the degree of smile of the useras the additional information, and

the second imaging unit images the second direction when the smile ofthe user is detected.

[14] The image processing apparatus according to [13], further includinga classification unit that records the second image in a recording areadetermined by the degree of smile, the degree of smile serving as theadditional information.

[15] The image processing apparatus according to [1], in which

the first imaging unit images a direction of a user operating the imageprocessing apparatus, the direction serving as the first direction,

the additional-information generating unit detects a smile of the userbased on the first image and generates the degree of smile of the useras the additional information, and

the image processing apparatus further includes a classification unitthat records the second image in a recording area determined by thedegree of smile, the degree of smile serving as the additionalinformation.

DESCRIPTION OF REFERENCE NUMERALS

-   -   11 image processing apparatus    -   21 display unit    -   22 in-image capturing unit    -   23 out-image capturing unit    -   53 signal processing unit    -   81 compression unit    -   82 additional processing unit    -   84 decoding unit    -   85 reproduction unit    -   121 face recognition unit    -   151 gaze-area detecting unit    -   181 viewing-angle converting unit    -   182 main-area detecting unit    -   183 crop processing unit

The invention claimed is:
 1. An image processing apparatus, comprising:a first imaging unit that is configured to image a first direction toobtain a first image; a second imaging unit that is configured to imagea second direction to obtain a second image, the second direction beingdifferent from the first direction; an additional-information generatingunit that is configured to generate additional information based on thefirst image to obtain a third image; a combining unit that is configuredto combine the third image with the second image to obtain a fourthimage, the fourth image having an aspect ratio that is the same as anaspect ratio of the second image; and a viewing-angle converting unitthat is configured to perform viewing-angle conversion processing on thesecond image, wherein, to generate the additional information based onthe first image, the additional-information generating unit is furtherconfigured to crop the first image by cutting out an area including amain subject on the first image, and wherein the combining unit isconfigured to combine the third image with the second image on which theviewing-angle conversion processing is performed.
 2. The imageprocessing apparatus according to claim 1, wherein, to generate theadditional information based on the first image, theadditional-information generating unit is further configured to compressan information amount of the first image.
 3. The image processingapparatus according to claim 2, wherein, to the generate the additionalinformation based on the first image, the additional-informationgenerating unit is further configured to compress a size of the firstimage.
 4. The image processing apparatus according to claim 3, wherein,to image the first direction to obtain the first image, the firstimaging unit is configured to image a direction of a user operating theimage processing apparatus.
 5. The image processing apparatus accordingto claim 1, wherein, to image the first direction to obtain the firstimage, the first imaging unit is further configured to image a directionof a user operating the image processing apparatus, and wherein, togenerate the additional information based on the first image, theadditional-information generating unit is further configured to performface recognition processing on the first image to generate photographerinformation of the second image as the additional information.
 6. Theimage processing apparatus according to claim 5, further comprising: anextraction unit that is configured to detect a specific image from aplurality of second images based on the additional information, thespecific image being captured by a specific photographer; and areproduction unit that is configured to display only the specific imagethat is detected by the extraction unit.
 7. The image processingapparatus according to claim 5, further comprising a classification unitthat is configured to record the second image in a recording areadetermined by the photographer information.
 8. The image processingapparatus according to claim 1, further comprising a display unit thatis provided to a surface facing a user operating the image processingapparatus, the first imaging unit being provided to the surface, whereinthe display unit is configured to display an image taken in by thesecond imaging unit at a time when the second image is captured, whereinthe additional-information generating unit is further configured todetect a line-of-sight direction of the user based on the first image,and wherein, to generate the additional information based on the firstimage, the additional-information generating unit is further configuredto generate gaze information from the first image, the gaze informationbeing indicative of a gaze area at which the user gazes on the secondimage.
 9. The image processing apparatus according to claim 8, furthercomprising a reproduction unit that is configured to control the displayunit to display the second image based on the additional information.10. The image processing apparatus according to claim 9, wherein thereproduction unit is further configured to superimpose and display aplurality of second images based on the additional information.
 11. Theimage processing apparatus according to claim 1, wherein, to combine thethird image with the second image to obtain the fourth image, thecombining unit is further configured to arrange and combine the thirdimage with the second image on which the viewing-angle conversionprocessing is performed, and generate a composite image as the fourthimage.
 12. The image processing apparatus according to claim 1, wherein,to image the first direction to obtain the first image, the firstimaging unit is further configured to image a direction of a useroperating the image processing apparatus, wherein, to generate theadditional information based on the first image, theadditional-information generating unit is further configured to detect asmile of the user based on the first image, and generate a degree ofsmile of the user as the additional information, and wherein, to imagethe second direction to obtain the second image, the second imaging unitis further configured to image the second direction when the smile ofthe user is detected by the additional-information generating unit. 13.The image processing apparatus according to claim 12, further comprisinga classification unit that is configured to record the second image in arecording area determined by the degree of smile.
 14. The imageprocessing apparatus according to claim 1, further comprising: aclassification unit configured to record the second image in a recordingarea determined by a degree of smile of a user operating the imageprocessing apparatus, wherein, to image the first direction to obtainthe first image, the first imaging unit is configured to image adirection of the user, wherein, to generate the additional informationbased on the first image, the additional-information generating unit isfurther configured to detect a smile of the user based on the firstimage, and generate the degree of smile of the user as the additionalinformation.
 15. An image processing method, comprising: imaging, by afirst imaging unit, a first direction to obtain a first image; imaging,by a second imaging unit, a second direction to obtain a second image,the second direction being different from the first direction;generating additional information based on the first image to obtain athird image; combining the third image with the second image to obtain afourth image, the fourth image having an aspect ratio that is the sameas an aspect ratio of the second image; and performing, by aviewing-angle converting unit, a viewing-angle conversion process on thesecond image, wherein generating the additional information based on thefirst image to obtain the third image further includes cropping thefirst image by cutting out an area including a main subject on the firstimage, and wherein combining the third image with the second imagefurther includes combining the third image with the second image onwhich the viewing-angle conversion process is performed.
 16. The imageprocessing method according to claim 15, wherein combining the thirdimage with the second image to obtain the fourth image further includesarranging and combining the third image with the second image on whichthe viewing-angle conversion process is performed, and generating acomposite image as the fourth image.
 17. A non-transitorycomputer-readable medium comprising a program that, when executed by aprocessing unit, causes the processing unit to perform a set ofoperations, the set of operations comprising: imaging, with a firstimaging unit, a first direction to obtain a first image; imaging, with asecond imaging unit, a second direction to obtain a second image, thesecond direction being different from the first direction; generatingadditional information based on the first image to obtain a third image;and combining the third image with the second image to obtain a fourthimage, the fourth image having an aspect ratio that is the same as anaspect ratio of the second image, wherein the set of operations furtherincludes performing, by a viewing-angle converting unit, a viewing-angleconversion process on the second image, wherein generating theadditional information based on the first image to obtain the thirdimage further includes cropping the first image by cutting out an areaincluding a main subject on the first image, and wherein combining thethird image with the second image further includes combining the thirdimage with the second image on which the viewing-angle conversionprocess is performed.
 18. The non-transitory computer-readable mediumaccording to claim 17, wherein combining the third image with the secondimage to obtain the fourth image further includes arranging andcombining the third image with the second image on which theviewing-angle conversion process is performed, and generating acomposite image as the fourth image.